WORLD INTELLECTUAL PROPEFHY ORGANIZATTON 
Internatiacial Bureau 




PCX 

INTCRNATIONAL APPLICATION PUBLISHED UNDER THE PATENT CXXJPERATION TREATY (PCT) 



(51) Intenatkma] Patent CtassiBcatitm ^ : 
G06K 9>00, 9/40, 9/34 



Al 



(11) Internatkmal PublkatiaD Numbv: WO 96A>9604 

(43) iDternatiaial PubUcstkm Date: 28 March 1996 (28i)3.96) 



(21) Intematkuial Applicatlao Number: PCT/US9S/1 1460 

(22) Internatkmal Flltng Date: 11 September 199S (1]J)9.9S) 



(30) Priority Data: 

08/309.061 



20 S^rtember 1994 (20i)9.94) US 



(71) AppUcant: NEOPATH, INC. [USAJSJ; 8271 154th Avenue 

NJB.. RedmoQd. WA 98052 (US). 

(72) iDTeaton: WILHELM. Paul. S^ 8217 122nd Avenue N3., 

Kirfcland, WA 98033 (US). LEE, Shifa-Joag. I4 14116 
Southeast 46tfa Street, Bellevue. WA 98006 (US). 

(74) Agent: LEONE, George, A^ Sr^ Leooe & MofTa, VJl., Suite 
1275. 601 Lakeshore Parkway. Minnetonka, MN 55305 
(US). 



(81) Dcstgnated States: AU. C^A, JP, Eun^iean patent (AT. BE, 
CH, DE. DK, ES. FR, GB, GR. IE, IT, LU. MC, NL. FT. 
SE). 



Published 

Wifh international search report. 
Before the expiration of the time limit for amending the 
claims and to be republished in the event of the receipt of 
amendments. 



(54) TitU: APPARATUS FOR AUTOMATED IDENXmCATION OF CELL GROUPINGS ON A BIOLOGICAL SPECIMEN 



(57) Abstract 

The detection (^cellular aggregates within cytologic samples. 
An image analysis system with an image gathering system (511) 
includes a camera (512), a motion controller (520). an illumination 
system (508) and an intufKe obtains images (55) of cell groupings. 
The image gathering system (511) is constmcted for gathering 
image data (38) of a specimen mounted on a slide and is coupled to 
a data pnx:essing system (540). Image data (38) is transfexred from 
the image gathering system (511) to the data processing system 
(540). The data processing system (540) obtains objects of interest. 
A four step process finds cellular aggregates. The tot step is 
acquisition of an image for analysis (12). The second step is 
extracu'an of image features (14). The third step is classification 
of the image (16) to determine if any potential cellular aggregates 
may exist in the image (24). The fourth step is segmentation of 
objects (18) which includes the substeps of detecting and locating 
potential cellular aggregates. 
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APPASATtrS FOR AUTOMATED IDENTI PI CATION OF CELL 
GROUPINGS ON A BIOIiOGICAI. SPECIMEN 

This invention relates to the automated detection 
of aggregates of cells within biologic san^lea such as 
cytologic specimens and more particularly to an 
automated cytology system that classifies cytological 
specimens, based in part on the analysis of aggregates 
of cells, as normal or needing human review, 
BACKGROUND OF THE INVENTION 
Historically, screening of cytologic material has 
been a task for trained technicians and 
cytopathologists . Even though screening is done by 
highly trained individuals, the task is repetitive and 
requires acute attention at all times. Since 
screening of cytologic material is repetitive and 
tedious, it, has been thought to be ripe for 
automation. On the other hand, the complexity emd 
variety of material found in cytologic specimens has 
proven very difficult to examine in an automated 
fashion. As a result, automated screening of 
cytologic specimens has been the unrealized goal of 
research for many years . 

Recent research has demonstrated an effective 
approach for detection and identification of cellular 
abnormalities as demonstrated by isolated cells in a 
cytologic sample (U.S. Patent Application Serial No. 
08/179,812, Method for Identifying Objects Using Data 
Processing Techniques) . However, in many cases, 
significant abnormalities are manifested within cell 
aggregates rather than as isolated cells. See 
"Diagnostic Cytopathology of the Uterine Cervix", by 
Stanley F. Patten, Jr. Identification of cell 
groupings in biological specimens has previously been 
achieved by human visual identification. It is 
important, therefore, that any automated screening 
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device have the capability of identifying 
abnormalities within cell aggregates as well as among 
isolated cells. Additionally, in cervical cytology, 
sample adequacy as defined in The Bethesda System "The 
5 Bethesda System for Reporting Cervical/Vaginal 
Cytologic Diagnoses", Robert J. Karman, Diane Soloman, 
Springer-Verlag, 1994 is determined, in part, through 
the identification of endocervical component cells in 
the sample that appear almost exclusively within 
10 aggregates. Therefore, the design of an automated 
screening device for cytologic samples must include 
the ability to detect and identify cellular 
aggregates . 

Therefore it is a motivation of the invention to 
15 automatically identify cell groupings within cellular 
specimens . 

SUMMARY OF THE INVENTION 

The invention provides a method and apparatus for 
the detection of cellular aggregates within cytologic 
20 samples. The invention comprises an image analysis 
system that further comprises an image gathering 
system having a camera, a motion controller, an 
illumination system, and an image transfer interface. 
The image gathering system is constructed for 

25 gathering image data of a specimen mounted on a slide. 

The image gathering system is coupled to a data 
processing system to transfer image data from the 
image gathering system to the data processing system. 
The data processing system analyzes the image data to 

30 identify objects of interest. The data processing 
system implements a four step process . The first step 
is the acquisition of an image for analysis. The 
second step is the extraction of image features. The 
third step is classification of the image to determine 

35 if any potential cellular aggregates may exist in the 
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image. The fourth step is segmentation of objects 
which include the substeps of detecting and locating 
potential cellular aggregates. 

Other objects, features and advantages of the 
5 present invention will become apparent to those 
skilled in the art through the description of the 
preferred embodiment, claims and drawings herein 
wherein like numerals refer to like elements. 

BRIEF DESCRIPTION OF THE DRAWIKQS 
^° illustrate this invention, a preferred 

embodiment will be described herein with reference to 
the accompanying drawings. 

Figures lA, IB and ic show an apparatus for 
automatic identification of cell groupings on a 
15 biomedical specimen. 

Figure 2A shows a process flow diagram of the 
image processing and analysis performed for each image 
of biologic specimens. 

Figure 2B shows a process flow diagram of the 
20 method of the invention to segment each image of 
biologic specimens . 

Figure 3 shows a process flow diagram for 
background object removal. 

Figures 4A, 4B, 4C and 4D show a schematic of a 
25 combination of two segmentation masks. 

Figures SA. SB, SC and 5D show process flow 
diagrams for the nuclear thresholding of the 
invention. Figure 5D further comprises Figure SE and 
Figure 5F which are intended to be read pieced 
0 together as a single figure. 

Figure 6 shows the process flow diagram for 
object refinement of the invention. 

Figures 7A and 7B show process flow diagrams for 
nuclei clustering of the invention. 
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DBTAXItSD DESCRXPTXON OP TEE PREFERRED EMBODIMENT 

In a presently preferred embodiment of the 
invention, the camera system disclosed herein is used 
in a system for analyzing cervical pap smears, such as 
that shown and disclosed in U.S. Patent Application 
Serial No. 07/838,064, entitled "Method For 
Identifying Normal Biomedical Specimens", by Alan C. 
Nelson, et al., filed February 18, 1992; U.S. Patent 
Application Serial No. 08/179,812 filed January 10, 
1994 which is a continuation in part of U.S. Patent 
Application Serial No. 07/83 8,3 95, entitled "Method 
For Identifying Objects Using Data Processing 
Techniques", by S. James Lee, et al . , filed February 
18, 1992; U.S. Patent Application Serial No. 
07/838,070, now U.S. Pat. No. 5,315,700, entitled 
"Method And Apparatus For Rapidly Processing Data 
Sequences", by Richard S. Johnston, et al . , filed 
February 18, 1992; U.S. Patent Application Serial No. 
07/838,065, filed 02/18/92, entitled "Method and 
Apparatus for Dynamic Correction of Microscopic Image 
Signals" by Jon W. Hayenga, et al . ; and U.S. Patent 
Application Serial No. 08/302,355, filed September 7, 
19 94 entitled "Method and Apparatus for Rapid Capture 
of Focused Microscopic Images" to Hayenga, et al., 
which is a continuation-in-part of Application Serial 
No. 07/83 8,063 filed on February 18, 1992 the 
disclosures of which are incorporated herein, in their 
entirety, by the foregoing references thereto. 

The present invention is also related to 
biological and cytological systems as described in the 
following patent applications which are assigned to 
the same assignee as the present invention, filed on 
September 20, 1994 unless otherwise noted, and which 
are all hereby incorporated by reference including 
U.S. Patent Application Serial No. 08/309,118, to Kuan 
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et al. entitled. "Field Prioritization Apparatus and 
Method, "U.S. Patent Application Serial No. 08/309.116 
to Meyer et al. entitled "Apparatus for Automated 
Identification of Thick Cell Groupings on a Biological 
Specimen." U.S. Patent Application Serial No. 
08/309,115 to Lee et al . entitled "Biological Analysis 
system Self Calibration Apparatus," U.S. Patent 
Application Serial No. 08/308,992. to Lee et al . 
entitled "Apparatus for Identification and Integration 
of Multiple Cell Patterns." U.S. Patent Application 
Serial No. 08/309,063 to Lee et al . entitled -A Method 
for Cytological System Dynamic Normalization.- U.S. 
Patent Application Serial No. 08/309,248 to Rosenlof 
et al. entitled "Method and Apparatus for Detecting a 
IS Microscope Slide Coveralip. " U.S. Patent implication 
Serial No. 08/309,077 to Rosenlof et al . entitled 
"Apparatus for Detecting Bubbles in Coverslip 
Adhesive." U.S. Patent Application Serial No. 
08/309,931. to Lee et al . entitled "Cytological Slide 
20 Scoring Apparatus. " U.S. Patent Application Serial No 
08/309.148 to Lee et al. entitled "Method and 
Apparatus for Image Plane Modulation Pattern 
Recognition," U.S. Patent Application Serial No. 
08/309,250 to Lee et al . entitled "Apparatus for the 
25 Identification of Free-Lying Cells," U.S. Patent 
Application Serial No. 08/309.209 to Oh et al . 
entitled "A Method and Apparatus for Robust Biological 
Specimen Classification," U.S. Patent Application 
Serial No. 08/309,117, to Wilhelm et al . entitled 
3 0 "Method and Apparatus for Detection of Unsuitable 
Conditions for Automated Cytology Scoring." 

Now refer to Figures lA and IB which show a 
schematic diagram of one embodiment of the apparatus 
of the invention for checking illumination integrity 
35 for an automated microscope. While the method and 
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apparatus of the invention will be discussed in terms 
of an example herein related to an automated cytology 
apparatus, it will be understood that the invention is 
not so limited. The features and principles of the 
invention may be applied to check urine analysis 
processes, semiconductor process defects, liquid 
crystal devices and other types of processing systems 
employing, for example, continuous arc lamps, filament 
lamps, laser sources, tube cameras, PIN diodes and 
photomultiplier tiibes . 

The apparatus of the invention comprises an 
imaging system 502, a motion control system 504, an 
image processing system 536, a central processing 
system 540, and a workstation 542. The imaging 
system 502 is comprised of an illuminator 508, imaging 
optics 510, a CCD camera 512, an illumination sensor 
514 and an image capture and focus system 516. The 
image capture and focus system 516 provides video 
timing data to the CCD cameras 512 , the CCD cameras 
512 provide images comprising scan lines to the image 
capture and focus system 516. An illumination sensor 
intensity is provided to the image capture and focus 
system 516 where an illumination sensor 514 receives 
the sample of the image from the optics 510 . In one 
embodiment of the invention, the optics may further 
comprise an automated microscope. The illuminator 508 
provides illumination of a slide. The image capture 
and focus system 516 provides data to a VME bus 538. 
The VME bus 53 8 distributes the data to an image 
processing system 53 6 . The image processing system 
536 is comprised of f ield-of -view processors 568. The 
images are sent along the image bus 564 from the image 
capture and focus system 516. A central processor 540 
controls the operation of the invention through the 
VME bus 53 8. In one embodiment the central processor 
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5S2 comprises a Motorola 68030 CPU. The motion 
controller 504 is comprised of a tray handler 518, a 
microscope stage controller 520, a microscope turret 
controller 522, and a calibration slide 524. The 
5 motor drivers 526 position the slide under the optics. 
A bar code reader 528 reads a barcode located on the 
slide 524. A touch sensor 530 determines whether a 
slide is under the microscope objectives, and a door 
interlock 532 prevents operation in case the doors are 
10 open. Motion controller 534 controls the motor 
drivers 526 in response to the central processor 54 0. 
An Ethernet (TM) communication system 560 communicates 
to a workstation 542 to provide control of the system. 
A hard disk 544 is controlled by workstation processor 
15 550. In one embodiment, workstation 542 may comprise 
a Sun Sparc Classic (TM) workstation. A tape drive 
54 6 is connected to the workstation processor 550 as 
well as a modem 548, a monitor 552, a keyboard 554, 
and a mouse pointing device 556. a printer 558 is 
connected to the Ethernet (TM) network system 560. 

During image collection integrity checking, the 
central computer 540, running a real time operating 
system, controls the automated microscope and the 
processor to acquire and digitize images from the 
microscope. The flatness of the slide may be checked, 
for example, by contacting the four corners of the 
slide using a computer controlled touch sensor. The 
computer 540 also controls the microscope stage to 
position the specimen under the microscope objective, 
and from one to 15 field of view (FOV) processors 568 
which receive images under control of the computer 
540 . 



20 



35 



Referring now to Figure IC, there shown is 
placement of a calibration and test target l into an 
optical path of an automated microscope 3 having a 
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turret 21. The calibration and test target may be 
mounted on a stage 521 substantially in a horizontal 
X,Y plane which intersects the optical path. The 
stage 521 is movable in the X,Y plane as well as along 
a Z axis which is perpendicular to the X,Y plane and 
which is parallel to the optical axis of the automated 
microscope. The turret 21 may comprise multiple 
objective lenses as is well known in the art. The 
microscope turret control 522 provides signals in a 
well known manner for positioning a selected objective 
lens into position for viewing a slide, for example. 

It is to be understood that the various processes 
described herein may be implemented in software 
suitable for running on a digital processor. The 
software may be embedded, for example, in the central 
processor 540. 

Refer now to Figure 2A which shows a process flow 
diagram of the method of the invention to analyze cell 
aggregates. An image is acquired in step 12. In a 
preferred embodiment of the invention, the image is 
acquired using a digital camera attached to a 
microscope as shown in Figures lA, IB and IC. The 
image acquired by the camera is of the cytologic 
specimen, magnified by an objective lens of 2 Ox 
magnification. The camera digitizes the image to 512 
by 512 pixels to a depth of 8 bits. The magnification 
of 20x and an image size of 512 by 512 pixels is by 
way of example and not limitation, and one skilled in 
the art will appreciate that other magnifications and 
image sizes may be used without departing from the 
scope of the invention. 

Since cellular aggregates may not exist in every 
acquired image, and since it is important to process 
images rapidly, the invention avoids extensive 
processing of images that contain no material of 
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interest. Image feature extraction 14 and image 
classification 16 show the process to achieve rapid 
removal of unproductive images. Features and 

properties of the image are measured in image feature 
5 extraction in step 14 . The measured features are then 
used to determine if the image may contain 
identifiable cellular aggregates. In one preferred 
embodiment, if a characteristic called 
AverageHighPixel value is greater than 240, then the 
10 image is rejected. AverageHighPixelValue may be 
defined as the average intensity value of all pixels 
with pixel counts above 200 in an image where 0 is 
black and 255 is white. The AverageHighPixelValue 
rule will identify images with very little cellular 
15 material. Such images have little chance of 

representing a cellular aggregate. 
Additionally, if : 

(SmaUDarkEdgeAverage * 35000) + HighPixelCount < 15000. 

then the image is rejected, where SmaUDarkEdgeAverage 
may be defined as the average value of image subject 
to a 5x5 closing residue operation, 

SmaUDarkEdgeAverage - 



20 



it'^Aui, iU^ « M) e ( 5x5 )) - /^) 

where N^^^, is the number of pixels in the image, 
Allpiaai, indicates that the summation covers all pixels 
in the image, 1^,^ is the original image, « is the 
morphological dilation operator (for example, as 
25 disclosed in Serra. j. , "image Analysis and 
Mathematical Morphology", Academic Press, 1982), e is 
the morphological erosion operator, closing residue is 
represented by the operation enclosed in parenthesis 
above, and HighPixelCount is the number of pixels with 



Mimosa V2-05-00 16:08:04 



pixel counts above 200 in the original image. The 
SmallDarkEdgeAverage rule will identify images with so 
much material that reliable detection and 
identification of cellular aggregates is unlikely. 

Refer now to Figure 2B which shows the image 
segmentation method of the invention. The image 
segmentation step 18, performs the identification of 
potential cellular aggregates. It is based on first 
identifying potential cell nuclei and then determining 
which nuclei lie close enough to other nuclei to be 
considered part of a cellular aggrregate. In one 
preferred embodiment, the image segmentation step 
includes five sub- steps. Segmentation steps 28 and 3 0 
remove background objects, segmentation step 32 is 
image thresholding, segmentation step 34 is object 
refinement, and segmentation step 36 is nuclei 
clustering. 

Refer now to Figure 3 which shows the method of 
the invention to remove large objects from the image. 
Since nuclei have a finite size range, it is useful to 
remove objects larger and smaller than that range 
before image thresholding step 32. In one preferred 
embodiment, large objects are removed by closing the 
image 3 8 with a 27 by 5 flat top structuring element 
40 and with a 5 by 27 flat top structuring element 42. 

foos.^ - nim[(( 7,^ e (27x5)) 9 (27x5)) . ((( /^^^ e (5x27)) e (5x27)))] 

The closed image 44 is then iteratively eroded 
conditioned on the original image until no more 
erosion takes place. This is termed ultimate 
conditional erosion 46. The structuring element for 
the ultimate conditional erosion is a flat top cross 
of 3x3 pixels. 
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15 



^ErodtCO) ^Closed 
{ 

{in^i » 1) = max [/o^. (/^^ e ((3x3) J 



1 

) 



In the above equations, a 3x3 cross structuring 
element is a center pixel, two adjacent horizontal 
pixels, and two adjacent vertical pixels, and where 
lErod.ii) represents the i"» iteration of conditional 
erosion equation in the while loop above, of the 
image. The residue 4 8 of the conditionally closed, 
eroded image and the original image contain only 
objects that are small enough to be nuclei, large 
objects are removed. The residue image is then opened 
with a 9 by 9 flat top structuring element 50 to 
remove objects smaller than valid nuclei thereby 
producing a residue image 52. 

To define potential nuclear regions of the image, 
the gray scale image is thresholded to produce a 
binary mask in step 32. In a binary mask, pixels may 
take on one of two values, active or inactive. Active 
pixels represent regions where potential nuclei have 
20 been identified. m one preferred embodiment. 

thresholding is done by combining results of two 
different methods, thereby incorporating the 
advantages of one to offset the disadvantages of the 
other. The first method segments the majority of 
25 nuclei completely with little problem of over 
segmentation. In one preferred embodiment, the nuclei 
identified by the first method is used when the second 
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method confirms that the mask of the first is not a 
false segmentation shown schematically in Figures 4A, 
4B, 4C and 4D, 

Now refer to Figures 4A, 4B, 4C and 4D, which 
show a graphical example of combination of two 
segmentation masks to take advantage of the strengths 
of each. In this preferred embodiment, the first 
segmentation mask is created by blurring, scaling, and 
clipping the original image 55 for use as a threshold 
image for the image that resulted from step 28 above. 
Blurring removes information other than background 
from the threshold image. Scaling provides 

appropriate threshold levels. Clipping assures that 
nuclei have at least a minimum residue strength before 
15 they are included in the mask . m a preferred 
embodiment, the functions are implemented as : 



10 



then the first mask is generated by the following 
rule : 

20 For each Pixel i 
{ 

Wuu^ > hoi^ then la^^ = 1 else 7^,,^ = 0 



Where liR^^aue is the i^^ pixel of image lR„idue and l^^iip 
is the i^** pixel of image Idip. In this preferred 
embodiment, the second mask is created by conditioning 
the residue image lR„idue. 

Often nuclei appear overlapped in an image. If 
a threshold is calculated for all nuclei without 
regarding overlap, nuclei that overlap will be 
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segmented as one. if the threshold for overlapping 
objects is adjusted to segment less, the nuclei may 
segment as separate objects. When objects overlap, 
their apparent darkness is greater; their residue will 
be stronger. Therefore, in one preferred embodiment, 
objects with strong residue are thresholded 
differently than those with weak residue. 

A modified residue image. containing 
only monotonically valued objects is created by 
conditionally dilating the residue image only in areas 
where pixels are greater than zero so that each object 
has all pixels equal to the value of the maximum pixel 
value for the object in the residue image. 

i = 1 

MvhileiisS) 

{ 



10 



For each Pixel j 



{ 

^/(W > 0) then I,,,,^^ = 



I + + 



An object is considered to have a strong residue 
if its maximum residue value is greater than 
20 StrongObjectTestValue where 
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StrongObjectTestVahie = 
{ImageBrightness * 3) + ImageEdgeStrength - 225 



ImageBrightness = .j^:^— (p^/s 
ImageEdgeStrength 



imagejuige:itrength = 

Where represents a binomial filter (in the case of 
a 3x3, it is a convolution operation, for which the 
kernel weights would be for the top row, for the 
middle row, and for the bottom row) . An image, 

is produced by retaining only the strong 
objects. For each Pixel i 

An edge image is created. 



From the edge image, the residue image, and the strong 
10 object image a threshold is computed. 

The mask for the second method is : 
For each Pixel i 

The nuclei mask is the conditional dilation of the 
15 second mask conditioned on the first mask (the 
conditional dilation is repeated 4 times to allow the 
complete dilation of all nuclei) . 
i=l 

while (i £ 4) 

20 

Now refer to Figure 5A which shows one embodiment 
of nuclear thresholding as employed in the method of 
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the invention. An original image 38 and a residue 
image 52 are fed to a first segmenter 172 and a second 
segmenter 174. The first segmenter 172 generates a 
first mask 176. The second segmenter 174 generates a 
second mask 178. The combination of the first mask 
176 and the second mask 178 in combiner 89 produces a 
nuclei image 124 . 

Now refer to Figure SB which shows the first 
segmenter of the invention. The original image 3 8 is 
fed to be blurred in 180. The image is then scaled in 
182. The image is then clipped in 184. The residue 
image 52 is then thresholded with the clipped image in 
186 to generate a first mask 176. 

Now refer to Figure 5C which shows the creation 
15 of the second mask 178. The original image 38 is fed 
to test the strong object value calculator 188. The 
residue image is fed to create a maximum residue image 
190 and an edge image 192. A strong object image is 
created by thresholding 194. The second mask 178 is 
20 created by taking the residue image 52, and 
thresholding with the threshold images, generated from 
the strong object image 194 and the edge image 192. 

Now refer to Figure 5D comprising Figure 5E and 
Figure 5F which are intended to be pieced together to 
show the processing flow for a preferred embodiment of 
nuclear thresholding to find a nuclei image from an 
original image, where a residue image has already been 
created from the original image. The cytological 
image processing method starts with the step of 
obtaining a digital representation of the cytological 
image 38. The invention then does a 3x3 erode 81 of 
the residue image 52 to provide a first representation 
101. A 3x3 dilation 82 of the residue image 52 
provides a second representation 102. Subtracting 83 
the residue image 52 from the second representation 
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102 provides a third representation 103, Subtracting 
84 the third representation 103 from the first 
representation 101 and setting all negative values to 
zero provides a fourth representation 104. The 
5 invention then compares the residue image 52 to zero 
to provide a binary condition control signal 125 . The 
invention then repeats a binary conditional dilation 
86 with a 3x3 cross, eight times to provide a fifth 
representation 105. The residue image 52 is 
10 transferred to a sixth representation 106 if the fifth 
representation 105 is greater than a nineteenth 
representation 119. The invention then 

morphologically computes 88 a binary result to a 
seventh representation 107, the binary result being 
15 one if the residue image 52 is greater than a 
predetermined combination of the fourth representation 
104, the residue image 52 and the sixth representation 
106, zero otherwise. In one embodiment, if the 
residue image 52 is greater than two times the fourth 
20 representation 104 plus 0.5 times the residue image 52 
plus .375 times the sixth representation 106, then the 
seventh representation 107 is set to a one, zero 
otherwise. A 3x3 blurring 91 of the original image 38 
provides an eighth representation 108. A 3x3 dilation 
25 92 of the original image 38 provides a ninth 
representation 109. A 9x9 erosion 94 of the original 
image 38 provides a tenth representation 110. The 
invention then subtracts the original image 3 8 from 
the eighth representation 108 to provide an eleventh 
3 0 representation 111. Subtraction 95 of the original 
image 3 8 from the ninth representation 109 provides a 
twelfth representation 112, Dilation 97 of the tenth 
representation 110 provides a thirteenth 
representation 113. Conversion of negative pixels to 
35 positive pixels of the same magnitude 98 for the 
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eleventh representation 111 gives the fifteenth 
representation 115. Computation of the pixel average 
99 of the twelfth representation 112 provides a 
fourteenth representation 114. Computation of the 
pixel average 79 of the fifteenth representation 115 
provides a seventeenth representation 117. 
Computation of the pixel average 78 of the original 
image 38 provides an eighteenth representation 118. 
Shifting 100 of the thirteenth representation 113 
right one bit provides a sixteenth representation 116. 
Computation of the sum 77 of the fourteenth 
representation 114, seventeenth representation 117, 
three times the eighteenth representation 118 and 
subtracting 255 provides the nineteenth representation 
119. Taking the maximum 75 of the sixteenth 
representation 116 and the value 10 provides a 
twentieth representation 120. Comparison 90 of the 
residue image 52 to the twentieth representation 120 
provides a twenty- first representation 121. 
Conditional dilation 89 of the seventh representation 
107 and twenty-first representation 121 provides the 
nuclei image 124 . 

Object refinement is conducted in step 34, Figure 
2B. Figure 6 shows the obj ect refinement step 34 in 
more detail. Small holes may be filled in 

segmentation by closing with a 3x3 structuring element 
126. The segmentation mask may then be smoothed by 
opening with a 5x5 structuring element 128. Note that 
the nuclei image is updated upon completion of the 
smoothing operation, creating the smooth nuclei image 



Now refer to Figure 7A which shows the 



130. 



NucUiNoHoUs 
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segmentation step for nuclei clustering. A nuclei 
image 124 has very small objects removed at 198 . 
Remaining objects are expanded depending upon their 
size where large objects are expanded more than small 
objects 199. A dilated image 159 is generated. 

Now refer to Figure 7B which shows processing 
flow for nuclei clustering. in one preferred 
embodiment, clustering 36 (Figure 2B) is nuclear size 
dependent. Small nuclei must be close to be 
considered part of an aggregate, while larger nuclei 
are not so restricted. Larger nuclei may be more 
distant and still considered part of an aggregate . 
Clustering is accomplished by dilating nuclei 
dependent on size. size dependent dilation is 
accomplished by creating nuclei masks for nuclei of 
different sizes, then dilating each according to size 
range and "OR"ing the masks to give a final cluster 
mask . 

husi = (W< © (5*5)) 
hiu2 = © ((3x5)c^)) 

where a 5x5 Diamond structuring element is: 

0 0 10 0 
0 1110 

11111 

0 1110 
0 0 10 0 
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and a 7x7 Diamond is the SxS Diamond dilated by a 3x3 
Cross. The invention takes the nuclei image 124 and 
does a 5x5 erosion 132 of the nuclei image 124 to 
provide a first cluster representation 151. A 3x3 
cross erosion 136 of the first cluster representation 
151 provides a second cluster representation 152. A 
3x3 cross erosion 138 of the second cluster 
representation 152 provides a third cluster 
representation 153. A 5x5 diamond dilation 140 of the 
third cluster representation 153 provides a fourth 
cluster representation 154. The logical ORing 142 of 
the second cluster representation 152 and fourth 
cluster representation 154 provides a fifth cluster 
representation 155. Dilating 144 the fifth cluster 
representation provides a sixth cluster representation 
156. Logical ORing 146 the first cluster 

representation 151 and sixth cluster representation 
156 provides a seventh cluster representation 157. A 
7x7 cross dilation 14 8 of the seventh cluster 
20 representation 157 provides an eighth cluster 
representation 158. A 4x4 dilation ISO of the eighth 
cluster representation 158 provides the segmented 
image 159. 

In one preferred embodiment, if objects in the 
segmented image 15 9 (1^^^^,, ) are larger than 24 00 
pixels, the object is considered to be a potential 
cellular aggregate. Other objects are removed from 
consideration . 

Referring again to Figure 2A, feature extraction 
30 20, is the measurement of features related to the 
segmented potential cellular aggregates. in one 
embodiment, the ratio of the standard deviation of the 
pixel values of the nuclei to the standard deviation 
of the pixel values of the cluster are measured. 
35 Also, the standard deviation of the nuclear 
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compactness is measured. Where the nuclear 

compactness is defined as: 

NuclearCompactness - (Perimeter ^2) /Area 
With feature values available the object 
5 classification 22 step may be performed. In one 
embodiment, an object is classified as probable 
squamous artifact if: 

(StdNuclei/StdCluster) + (StdNucCorapact * 0.038) > 
1.14. 

10 The invention has been described herein in 

considerable detail in order to comply with the Patent 
Statutes and to provide those skilled in the art with 
the information needed to apply the novel principles 
and to construct and use such specialized components 

15 as are required. However, it is to be understood that 
the invention can be carried out by specifically 
different equipment and devices, and that various 
modifications, both as to the equipment details and 
operating procedures, can be accomplished without 

20 departing from the scope of the invention itself. 
What is claimed is: 
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1- An automated cell group recognizer (lO) for 
locating groups of cells on a biological 
specimen comprising: 
S (a) an automated microscope (5li) for 

acquiring an image representation (12) 
of a portion of the biological specimen 
wherein the automated microscope (511) 
has an image representation output; 
^ image feature extractor (14) coupled 
to the automated microscope (5li) to 
receive the image representation, 
wherein the image feature extractor 
(14) has an image feature vector 
output; and 
(c) a classifier (16) coupled to the image 
feature vector output, wherein the 
classifier (16) has an output 
indicative of a likelihood that a 
predetermined part of the biological 
specimen contains a group of cells 
wherein the classifier (16) also has a 
likelihood output. 



20 
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2. The automated cell group recognizer of claim 
1 further including a segmenter (18) 
connected to the image representation 
output, the image feature vector output, and 
the likelihood output where the segmenter 
(18) has an output that locates cell groups 
the biological specimen. 

3. The apparatus of claim 2 further including 
an object feature extractor (2 0) connected 
to the group location output, feature vector 
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output and image representation having a 
group feature vector output . 

The apparatus of claim 3 further including 
an object classifier (22) connected to the 
group feature vector output, image feature 
vector output wherein the object classifier 
(22) has an object classification output. 

The apparatus of claim 1 wherein the 
biological specimen is a pap smear. 

The apparatus of claim 1 wherein the 
biological specimen is a gynecological 
specimen. 

The apparatus of claim l wherein the group 
of cells (55) comprise cells that overlap. 

The apparatus of claim 1 wherein the portion 
part of the biological specimen comprises at 
least one field of view of the automated 
microscope (511) . 

The apparatus of claim 1 wherein the image 
representation further con^rises pixels and 
wherein feature extractor (19) further 
comprises : 

(a) a pixel intensity averager (562) 
connected to the image representation 
output wherein the pixel intensity 
averager has an average pixel output 
for pixels having an intensity greater 
than a first predetermined intensity; 

(b) a small dark averager (562) connected 
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to the image representation output 
having a small dark averager output; 
ajid 

(c) a high pixel counter (562) connected to 
the image representation to count a 
number of pixels greater than a second 
predetermined intensity wherein the 
high pixel counter has a count output . 

The apparatus of claim 9 wherein the first 
predetermined intensity is at least 200 for 
an 8 bit pixel . 



The apparatus of claim 9 wherein the small 
dark averager further comprises a computer 
processor (562) coupled to the image 
representation output to compute a small 
dark average by the following equation: 

SmallDarkEdgeAverage = 

The apparatus of claim l wherein the 
classifier (16) further comprises an average 
high pixel value comparator (562) to compare 
an average high pixel value against a first 
predetermined intensity threshold wherein 
the average high pixel value comparator 
outputs the likelihood output. 

The apparatus of claim 1 wherein the 
classifier (16) further comprises a small 
dark averager/high pixel comparator (562) 
connected to the image representation output 
wherein the small dark averager/high pixel 
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comparator outputs the likelihood output. 

14. The apparatus of claim l wherein the 
classifier (16) further comprises a computer 
processor (562) coupled to the image 
representation output to compute a small 
darJc average by the following equation: 

SmallDarkEdgeAverage = 

7^ (({ « (5^) © ( 5x5 )) - /^) . 

15. The apparatus of claim z wherein the 
classifier (16) further comprises a small 
dark averager/high pixel comparator 
connected to the image representation . 
wherein the small dark averager/high pixel 
comparator further comprises a processor 
(562) where image representation output is 
used to compute a small dark average by the 
following equation: 

SmallDarkEdgeAverage = 
^ aT^ Hi lor^ « (5x5)) e ( 5x5 )) - /^,, . 

.6. The apparatus of claim 2 wherein the 
segmenter (18) further comprises: 

(a) a large object remover (28) to filter 
objects greater than a first 
predetermined size from the image 
representation output having a first 
image output ; 

(b) a small object remover (30) connected 
to the first image output to filter 
objects smaller than a second 
predetermined size having a second 



Mimosa V2-05-00 16:08:04 



wo 96/09604 

PCT/US95/11460 



(C) 



15 



- 25 - 
image output; 

an image thresholder (32) connected to 
the second image output to locate a 
nuclei -like object having a third image 
5 output ; 

(d) an object refiner (34) to fill small 
holes and remove jagged edges from the 
third image output having a fourth 
image output; and 

a cluster (36) connected to the fourth 
image output to group nuclei that are 
in close proximity aijd eliminate those 
that are not in close proximity wherein 
the cluster (36) has an output that 
locates all groups on the biological 
specimen. 

17. The apparatus of claim 16 where the large 
object remover (28) and small object remover 
(30) comprise grey scale morphological 
20 operators. 

IB. The apparatus of claim 16, wherein the large 
object remover (28) further comprises a 
field of view computer (568) . 

19. The apparatus of claim 16 wherein the small 
object remover (30) further comprises a 
morphological processor (88) to perform an 
opening operation. 

20. The apparatus of claim 19 wherein the 
opening operation comprises a 9x9 flat top 

30 operation (94) . 
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An image segmenting apparatus (18) for 
segmenting an image comprising: 

(a) a large object remover (28) to filter 
objects greater than a first 
predetermined size from the image (55) 
having a first image output; 

(b) a small object remover (3 0) connected 
to the first image output to filter 
objects small them a second 
predetermined size having a second 
image output; 

(c) an image thresholder (32) connected to 
the second image output to locate a 
nuclei-like object having a third image 

output ; 

(d) an object refiner (34) to fill small 
holes and remove jagged edges from the 
third image output having a fourth 
image output; and 

(e) a clusterer (36) connected to the 
fourth image output to group nuclei 
that are in close proximity and 
eliminate those that are not in close 
proximity. 

A method of creating a residue image (52) 
from a digital image (3 8) comprising the 
steps of : 

(a) first closing the digital image to 
provide a first representation (4 0) ; 

(b) second closing the digital image to 
provide a second representation (42); 

(c) minimizing the first representation and 
second representation to provide a 
third representation (44) ; 
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(d) ultimately conditionally eroding with 
a 3x3 cross, the third representation 
(46) ; 

(e) svibtracting the third representation 
from the digital image to provide a 
fourth representation (48); and 

(f) opening the fourth representation (50) 
to provide the residue image (52) . 

The method of claim 22 wherein step (a) , 
first closing the digital image, comprises 
a 5x27 closing (40) . 

The method of claim 22 wherein step (a) , 
first closing the digital image, comprises 
a 27x5 closing (42) . 

A cytological image processing method (Fig. 
5D) for finding a nuclei image in an image 
(55) , where a residue image (52) has already 
been created, the cytological image 
processing method (Fig. 5D) comprising the 
steps of : 

(a) obtaining a image representation of the 
image (52) ; 

(b) 3x3 eroding the residue image to 
provide a first representation (81) ; 

(c) 3x3 dilating the residue image to 
provide a second representation (82) ; 

(d) subtracting the residue image from the 
second representation to provide a 
third representation (83); 

(e) subtracting the third representation 
from the first representation and 
zeroing negative pixels to provide a 
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fourth representation (84) ; 

(f) comparing the residue image to zero 
(85) to provide a binary condition 
control signal (125) ; 

(g) repeating a binary conditional dilation 
with a 3x3 cross, eight times to 
provide a fifth representation (86); 

(h) providing the residue image to a sixth 
representation if the fifth 
representation is greater than a 
nineteenth representation (87) ; 

(i) morphologically computing a binary 
result to a seventh representation, the 
binary result being one if the residue 
image is greater than a predetermined 
combination of the residue image, the 
fourth representation, and the sixth 
representation, zero otherwise (88) ; 

( j ) 3x3 blurring the image to provide a 

eighth representation (91) ; 
(k) 3x3 dilating the image to provide a 

ninth representation (92) ; 
(1) 9x9 eroding the image to provide a 

tenth representation (94) ; 
(m) subtracting the image from the eighth 

representation to provide an eleventh 

representation (95) ; 
(n) subtracting the image from the ninth 

representation to provide a twelfth 

representation (96) ; 
(o) dilating the tenth representation to 

provide a thirteenth representation 

(97) ; 

(p) computing an absolute value of the 
eleventh representation to provide a 
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(s) 



fifteenth representation (98) ; 
(q) computing a pixel average of the 
twelfth representation to provide a 
fourteenth representation (99) ; 
(r) computing a pixel average of the 
fifteenth representation to provide a 
seventeenth representation (79); 
computing a pixel average of the image 
to provide an eighteenth representation 
10 (78); 

(t) shifting the thirteenth representation 
right one bit to provide a sixteenth 
representation (100) ; 

(u) computing a sum of the seventeenth 
representation, fourteenth 
representation, three times the 
eighteenth representation and 
subtracting 255 to provide the 
nineteenth representation (77) ; 
taking a maximum of the sixteenth 
representation and the value lo to 
provide a twentieth representation 
(75) ; 

comparing the residue image (52) to the 
twentieth representation to provide a 
twenty-first representation (90) ; and 
conditionally dilating the seventh 
representation and twenty- first 
representation (89) to provide the 
^0 nuclei image (124). 

26. The cytological image processing method 
(Fig. 5D) of claim 25 further comprising the 
steps of closing (126) the nuclei image 
(124) to provide a closed image and opening 



(w) 



(x) 
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the closed image (128) to provide a smoothed 
nuclei image (130) . 

27 . A method of segmenting an image into a 
segmented image (36) , the method comprising the 
5 steps o£: 

(a) 5x5 eroding a nuclei image (124) to 
provide a first representation (132); 

(b) 3x3 cross eroding the first 
representation to provide a second 

10 representation (13 6) ; 

(c) 3x3 cross eroding the second 
representation to provide a third 
representation (138) ; 

(d) 5x5 diamond dilating the third 
15 representation to provide a fourth 

representation (140) ; 

(e) logical ORing the second representation 
and fourth representation to provide a 
fifth representation (142) ; 

20 (f) dilating the fifth representation to 

provide a sixth representation (144) ; 
(g) logical ORing the first representation 
and sixth representation to provide a 
seventh representation (146) ; 

25 (h) 7X7 cross dilating the seventh 

representation to provide an eighth 
representation (148) ; and 
(i) 4X4 dilating the eighth representation 
(150) to provide the segmented image 

30 (159) . 

28 . A method of nuclear thresholding comprising 
the steps of : 

(a) acquiring an original image of a 
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biological specimen (38) ; 

(b) computing a residue image of the 
original image (52); 

(c) performing a first segmentation method 
5 on the original image and the residue 

image (172) ; 

(d) performing a second segmentation method 
on the original image and the residue 
image (174) ; and 

(e) combining results of the first 
segmentation method and second 
segmentation method (89) to identify 
nuclei (124) of objects on the 
biological specimen . 

15 29. The method of claim 28 wherein the first 

segmentation method (172) comprises: 

(a) blurring the original image to form a 
blurred image (180) ; 

(b) scaling the blurred image to form a 
20 scaled image (182) ; 

(c) clipping the scaled image to form a 
clipped image (184); and 

(d) thresholding (186) the residue image 
(52) with the clipped image (184) to 

25 generate a first mask (176) that is 

combined with an output of the second 
segmentation method to identify nuclei 
of cells on the biological specimen. 

30. The method of claim 28 wherein the second 
3 0 segmentation method (174) comprises the steps of: 

(a) calculating a strong object test value 
from the original image (188); 

(b) creating a maximum residue image (190) 
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from the residue image (52) ; 

(c) creating an edge image from a clipped 
image (192) / 

(d) thresholding a maximum clipped image 
5 with the strong object test value to 

generate a strong object image (194); 

(e) creating a threshold image from the 
strong object image, the edge image, 
and residue image (196) ; and 

10 (f) generating a second mask that is 

combined with an output of the first 
segmentation method to identify nuclei 
in the original image of the biological 
specimen (178) . 

IS 31. A method of nuclei clustering (Fig. 7A) 

comprising the steps of: 

(a) removing very small objects from a 
nuclei image (198) ; 

(b) expanding remaining objects depending 
2 0 upon size where large objects are 

expanded more than small objects (199) 
to provide an expanded output and a 
nuclei clustered output (159) . 
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